In the medical treatment area, it is usually necessary to utilize a number of modalities to view the internal anatomy of the person to be treated. These modalities include X-ray imaging, Magnetic Resonance Imaging ("MRI"), and computed tomography ("CT") imaging. There also are other modalities such as functional MRI ("fMRI"), single photon emission computed tomography ("SPECT"), and positron emission tomography ("PET"), all of whose images contain physiologic or metabolic information depicting the actions of living tissue.
It is known that each of modalities has certain strengths and weaknesses in the images that it produces. For example, X-radiography ("X-ray") imaging has high spatial and intensity resolutions, shows bony anatomy with high detail, and is relatively inexpensive to use; but X-ray also presents the viewer with complex two-dimensional ("2-D") views of superimposed anatomy. X-radiography also has difficulty resolving soft tissue features. MRI has the advantage of displaying three-dimensional ("3-D") images of soft tissues with high contrast and high spatial resolution, but does not image bone well. CT imagery, based on X-ray absorption, produces 3-D pictures of bony anatomy, and, increasingly, good definition of soft tissue, although MRI remains the preferred modality for viewing soft tissue. However, if the correct modalities are selected and aligned, the resulting combined image data will provide a more complete representation of the internal anatomy of the patient. Given this, the issue is to identify a system and method that will align image data from two modalities very accurately in a reasonable amount of time.
Image alignment, which is part of the science of image fusion, has been used since at least World War II when time course aerial photography was used to support strategic bombing. Blink comparator machines were used to superimpose two images taken at different times by displaying them to the viewer in rapid succession. As these images were being displayed in this manner, their features were matched to quantify the bombing damage.
This same basic comparison technique has been used to superimpose images produced by remote sensors, for example aerial or satellite sensors, but with the refinement of using computer-based methods to align images produced at different times, or at the same time by different channels of the sensor. These images were digital images. The techniques just described for image alignment in remote sensors were reported in Castleman, K. R., "Digital Image Processing" Prentice-Hall. Englewood Cliffs, N.J., 1979, and Moik, J. G., "Digital Processing of Remotely Sensed Images," NASA SP-431, Washington, D.C., 1980.
The use of computers to effect image alignment became a necessity as the number of images produced by LANDSAT, SPOT, and other satellite systems rapidly rew and there was the need to perform nonlinear warping transforms to match the images taken from different sky-to-ground perspectives. The use of computers also became a necessity to effect alignment of brain tissue images so that useful and effective neurophysiological research could be conducted. This was reported in Hibbard, L, et al., Science, 236:1641-1646, 1987. Reviews of various computed alignment techniques have been reported in Brown, L. G., "A Survey of Image Registration Techniques," ACM Computing Surveys, 24: 325-376, 1992 and Van den Elsen, P.A., et aL, "Medical Image Matching--A Review with Classification," IEEE Engineering in Medicine and Biology, March 1993, pp. 26-39.
Image fusion of images from at least two modalities is currently being used in radiation onocology because it has been found to provide better tumor definition, which was reported in Rosenman, J. G., et aL, "Image Registration: An Essential Part of Radiation Therapy Treatment Planning," International Journal of Radiation Onocology, Biology, and Physics, 40:197-205, 1998. It is anticipated that there will be increased development of software tools for the use in radiation treatment planning ("RTP") to effect image alignment and display/contouring tools to process and use the fused images that are produced.
Image alignment methods generally fall in two categories: manual alignment by direct inspection of the images being aligned, and automatic alignment by computing a solution to a numerical problem with some type of computer program. Manual alignment ("MA") is carried out visually by matching corresponding features of two images. MA may be implemented in all RTP systems that offer or purport to offer image fusion The mechanism that most frequently is used in MA is the visual placement of fiducial points or markers from which a transformation is derived, for example, by a least-square minimization of the differences between corresponding landmark points. A method of implementing MA that is not as frequently used is the visual matching of objects imbedded directly in the images being aligned.
MA is usually carried out using a graphical user interface ("UGU"). In most cases, the GUI simultaneously displays the axial, sagittal, and coronal plane views of an area of interesl The GUI must provide efficient navigation of 3-D data, and precise localization of fiducial markers or other tools for alignment or measurement Heretofore, commercial systems have used UNIX workstations or special-purpose computers to achieve the high computation and graphics throughput needed to interactively display large data volumes
Automated image alignment ("AIA") methods involve obtaining a transformation through computation of the properties of the images to be registered. This may even take place through programmed computer actions without user intervention. Many of the more successful AIA methods are based on correlation and moment invariants matching.
Correlation methods, more accurately, involve the cross-correlation or cross-covariance of image pixel intensities. These methods produce robust and accurate alignments of images. Methods of this type have been reported in Anuta, P. E., "Spatial Registration on Multispectral and Multitemporal Digital Imagery Using Fast Fourier Transform Techniques," IEEE Transactions on Geoscience Electronics, 8:353-368, 1970; Bamea, D. I., et al., "A Class Of Algorithms For Fast Digital Image Registration," IEEE Transactions on Computers, 21:179-186, 1972; and Pratt, W. K., "Correlation Techniques of Image registration," IEEE Transactions on Aerospace and Electronic Systems, 10:353-358, 1974.
According to most correlation methods, the location of the correlation peak will correspond directly to the translation needed to align images that already have a correct rotational alignment. The rotational alignment may have been obtained by correlating the images after resampling them on a polar grid according to the method reported in Hibbard, L, et al., Science 236:1641-1646, 1987. This rotational alignment method depends on the use of a fast Fourier transform. Correlation methods are accurate and robust to noise and differences in image content as long as the images are not too different. Accordingly, correlation methods are most often used for the alignment of images of the same kind.
Moment invariants methods are computed methods for image registration. A example of these methods is reported in Jain, A. K., "Fundamentals of Digital Picture Processing, Prentice-Hall Englewood Cliffs, N.J., 1989. This moment invariants method involves the use of the principal moments computed from a two-dimensional ("2-D") inertia matrix of some prominent object of the images. The principal moments correspond to a unique set of principal vectors. Image alignment is performed by transforming one image's principal vectors onto the other image's principal vectors. This usually is a fast calculation. However, the moment invariants method depends on the ability to efficiently extract the object or set of objects that serve as the moment fiducials. This method has considerable problems if the images are complex and have no dominant single feature for the alignment of images of the same kind or modality.
Besides the moment invariants method just discussed, there are other alignment methods based on invariant properties. These other methods may use geometric invariants, and minima/maxima of curvature and other extrema. At least one of these methods is reported in Lavellee, S., "Registration for Computer Integrated Surgery: Methodology, State of Art, in Taylor, R., et al, Computer Integrated Surgery, MIT Press, Cambridge, Mass., 1996. The characterization of these methods are that they are fast but susceptible to noise and inter-subject differences. These systems, however, only use a small portion of all of the available information from each image set to compute the registration, which has some effect on the accuracy.
In a practical sense, the automated alignment of CT and MRI images of the head was accomplished in the late 1980's and early 1990's. The methods that were used at that time gained some level of use but they had drawbacks. These methods were reported in Pelizzari, C. A. et al., "Accurate Three-dimensional Registration of CT, PET, and/or MR Images of the Brain," Journal of Computer Assisted Tomography, 13:20-26, 1989.
The method described in Pellzzari, et al. suggests representing the surface of the brain in CT and MRI by stacks of contours of the cortical surface and minimizing the least-squares differences between neighboring points in the two surfaces to determine the alignment transform. This method was later refined by sub-sampling the surface points and then applying simple optimization to determine the transformation. This late refinement was reported in Van Herk, M. and Kooy, H. M., "Automatic Three-Dimensional Correlation of CT--CT, CTMRI, and CT-SPECT Using Chamfer Matching," Medical Physics, 21:1163-1178, 1994. It was found that the original and refined methods were useful only for the head, and both methods required that the cortical surface be manually contoured first.
Another automatic method that has been suggested for the registration of images is based on the maximization of mutual information and this method was first reported on in 1995. This method has its genesis in information theory in which the relatedness of one random variable for another is based on the measure of the variables' entropies, which is referred to as mutual information ("MI").(See, Cover, T. and Thomas, J, Elements of Information Theory, John Wiley and Sons, New York, 1991). Thus, for two images, the MI is small if the images are unrelated, or related but unregistered. If the images registration is improved, their MI increases and is maximized when the images are geometrically aligned The MI is different from correlation in that systematic differences between images which would confound correlation actually strengthen the alignment by MI. MI is frequently used for two- and three-dimensional alignment of multimodal imagery in medical science as reported in Wells, W. M., et al., "Lecture Notes In Computer Science," Vol. 1496, Springer-Verlag, N.Y., 1998.
The use of MI has also been the subject of a number of studies. For example, it has been used for aligning multimodal MRI and MRI with CT as reported in Viola, P. and Wells, W. M., "Alignment by Maximization of Mutual Information," Proc. of the Vth Int'l. Conf. on Computer Vision, Cambridge, Mass., June 1995, pp. 16-23; Collignon, A., Vandermeulen, D, et. al., Automated Multimodality Medical Image Registration Based On Information Teory," in Bizais, Y., et al., Proc. of the XVth Int'l. Conf. on Computer Vision, Virtual reality, and Robotics in Medicine (CVRMed '95), Vol 905, Springer-Verlag, Lecture Notes in Computer Science, Nice, France, April 1995, pp. 263-274; Meas, F., et al., "Multi-Modality Image Registration by Maximization of Mutual Information," IEEE Workshop on Mathematical Methods in Biomedical Image Analysis, San Francisco, Calif., Jun. 21-22, 1996, pp. 14-22; and Studholme, C., et al., "Automated 3-D Registration of MR and CT Images of the Head," Medical Image Analysis, 1:163-175, 1996. Among other things, the issue with MI, however, is the ability to perform it in a time effective manner.
In 1997, the Vanderbilt Retrospective Brain Registration study was conducted. This study was directed to computed alignment methods with the goal of assessing the error of each of the standard group of head data sets comprising CT+MRI or PET+MRI data sets in which "gold standard" alignments (alignments by which the others were judged) were measured using bone-implanted markers. In the study, evidence of the markers were removed from the images before being sent to the participating laboratories. The techniques used by the testing laboratories included variations of the surface matching, correlation, and MI. The errors in respective results varied greatly. There were at least fourteen methods used to align CT and MRI images. Two of the laboratories that used MI achieved the lowest total median error over the spectrum of MRI pulse-sequence modalities. The third best results were achieved by a correlation method. In a comparison of alignments using PET and MRI, the first and second lowest total median errors were achieved by a single correlation method followed by two MI methods. In summary the MI and correlation methods produced the lowest errors and did not require brain contouring, scalp removal, or other interactive activities. Although these methods showed some success, it is believed that the time to solution could be very much improved without sacrificing accuracy.
The present invention provides a system and method that includes an automated alignment method that is accurate but much faster that the prior art systems and methods; and the system and method of the present invention includes an interactive alignment method that achieves speeds and displayed detail equal to, or exceeding, the highest performance level of the prior art, without the need of specially configured or extremely powerful high performance workstations.